Electronic image compositing is employed by the video industry for superimposing (compositing) a foreground (FG) subject over a background (BG) scene. The FG subject is typically a person or an object placed before an illuminated colored backing. The compositing apparatus replaces the colored backing with the BG scene. This process is used extensively in producing TV shows and commercials.
For nearly 30 years a switching process known as chroma key has been used for compositing video images. Switching occurs between the BG scene and the FG subject as a function of the presence or absence of the chroma of the backing.
A switching compositing system which switches as a function of the presence or absence of the chroma of the backing represents an "OR" logic; that is, one sees either the FG scene "OR" the BG scene but not both. When a subject is semitransparent, such as thin smoke, liquids and glassware, one should see the subject, and one should also see the background scene through the subject. For semitransparent subjects, there is no appropriate place to switch. A switching compositing system is incapable of reproducing semitransparent subjects. More recently the applicant has introduced to the video industry a linear, non-switching, image compositing method utilizing an "AND" concept. Apparatus which utilize this process are sold under the trademark Ultimatte. In this method a linear control signal (E.sub.c) is formed as a function of the brightness and visibility of the colored backing. Any color may be used for the backing, but blue has been most popular, and therefore the colored backing is often referred to as the Blue Backing. The backing will therefore be referred to as a blue backing in the description of this invention.
The linear control signal E.sub.c is normally used to reduce the chroma and luminance of the backing to a low level such as neutral grey or to zero video (black). E.sub.c also linearly controls the level of the BG scene video. The BG scene is thus visible through the FG subject to the extent the subject was transparent. Since the colored backing may be reduced to black, the BG scene may be added to the FG subject by simple addition. The linear compositing method is described by the applicant's following U.S. Pat. Nos. 3,595,987; 4,007,487; 4,100,569; 4,344,085; 4,409,611; 4,589,013; 4,625,231. The linear (non-switching) compositing method produces superb and undetectable composite images. Smoke, liquids, glassware and fine detail, such as individual strands of hair, are composited as if the scene were real. Shadows are retained and become shadows on the BG scene. Unfortunately, the linear compositing method also reproduces every wrinkle in the backing, its texture, its seams, smudges, rub marks and unintentional shadows.
On large colored backings it is not a simple matter to achieve uniform illumination. Large backings require multiple light sources whose patterns imperfectly overlap. The light sources must be kept out of the camera field thus placing them in positions not conducive for achieving illumination uniformity. Limitations of available electric power, or an insufficient number of lighting units, typically results in lower levels of illumination to be compensated for by opening the camera lens aperture a stop or two. Even if the backing illumination were uniform, a large aperture setting on the camera lens induces some lens vignetting (corner fall off).
For a variety of reasons the colored backing is not likely to provide a uniform video signal level over the entire field. Since E.sub.c is derived from this colored field and controls the level of the BG scene, unless E.sub.c is corrected, the background scene will contain all of the brightness variations occurring on the colored backing and induced by lens vignetting.
A clean, smooth, seamless backing having uniform luminance is sometimes achieved, but this ideal is not always practical, nor economically feasible. In applicant's later patents, methods and apparatus are disclosed for correcting backing defects, nonuniform illumination, and undesired shadows by electronic means. For example, an adjustable E.sub.c clip level causes the backing to appear to be uniformly illuminated. A second circuit called "cleanup" helps to remove backing seams as well as dusty footprints tracked onto a clean blue floor. When used for small corrections, the E.sub.c clip and cleanup do not produce an observable penalty.
The penalty for using substantial amounts of E.sub.c clipping is that the subject appears to be backlighted. This backlighting effect causes a slight glow to loose hair and to subject edges. While a backlight effect may sometimes be desirable, in other scenes it is not acceptable.
The penalty for using excessive cleanup is the loss of fine edge detail, such as individual strands of hair. The loss of edge detail is quite essential in eliminating the fine wires used to "Fly" objects or persons. However, when cleanup is used to eliminate footprints or backing defects, loss of hair detail becomes an unwelcome tradeoff.
The preceding discussion dealt with luminance nonuniformity of the backing. A second type of nonuniformity is color. One part of the backing will appear to have a slightly different shade of blue, for example. Although one may use the identical material in coloring a wall and floor, the wall will always have a higher color saturation than the floor. Visually, and to the camera, a wall appears to be a bright blue, while a floor appears to be a bluetinted grey. The camera faces the wall surface at essentially right angles, but the floor is seen at a small low angle. Diffuse surfaces provide low angle scattering of white light and thus the floor appears to be a somewhat desaturated blue.
When one subtracts a sufficient level of the E.sub.c control signal from FG video to cause the blue wall to go black, the floor remains a dull grey. Additional subtraction of the control signal to reduce the floor to black will cause the wall to approach black too soon causing a darkened edge to FG subjects in the wall area.
Applicant's U.S. Pat. No. 4,625,231 compensates for the wall/floor color difference by using an automatic bias circuit. This circuit continuously monitors the video in the blue backing area and adjusts the E.sub.c control signal level on a line-by-line basis to result in video just reaching zero in both the wall and floor areas.
This method of correcting for color variation works well for wall and floor color differences, but it does not adequately correct for color differences that occur for blue set pieces. A styrofoam rock, a flat, a stairway or other shape is painted blue and placed on the stage to match the size, shape and apparent position of similar objects in the BG scene. Such blue set pieces add realism to the composite scene, since the actor's shadow is distorted by the rock or staircase, as it would have been had he been in the actual BG scene. Set pieces also permit the actor to appear to walk behind objects that are actually in the BG scene.
These blue set pieces may be placed at various angles and positions with respect to the camera and set lighting lamps. The result is a slight color difference. An autobias circuit cannot adequately correct for color and brightness differences on individual set pieces.